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A Taxonomy of Evaluation Models: 
Use of Evaluation Models in Program Evaluation 

Wayne E. Carter 

4)allas Independent School District 

Intraduction 

In 1965 the federal government continued its support of 
constructive changes in education by passage of Public Law 89-10: 
The Elementary and SecDndary Education Act of 1965# This law 
explicitly stated that school systems and schools receiving funds 
through this Act were required to furnish annual evaluations of 
their projects to state and federal agencies. 

The first annual report for Title I of the Elementary ari'd 
Secondary Education Act indicated that competent evaluation per- 
sonnel were not generally available in the local school systems at 
that tifne. This first report, evaluating thousands of Title I 
projects, was almost entirely anecdotal in nature a^.a nearly 

devoid of statistical data in support of the success of the 
1 

various projects. 

To meet the needs created by the passage oi this Act, 

evaluation centers were created in various parts of the country. 

One of these centers was opened at The Ohio State University in 

Columbus, Ohio, under the direction of Dr. Daniel L. Stuff lebeam. 

2 

Stufflebeam and his staff,* in examining th^ first report 
of the Title I projects, determined that the ae irth of adequate 
evaluation information was due to several problems including the 
lack of trained evaluators, adequate evaluation theory, and of 



adequate instruments and procedures for evaluation. Furthennore , 
they believed that the following three types of conceptual prob- 
lems prevented provision of effective evaluation: 

1- A lack of understanding of decision processes and in- 

-formation requirements in current programs of educational 
' change; ' 

2. The lack of a definition of educational evaluation 
pertinent to emergent requirements for educational 
evaluation; and 

3 

3. A lack of appropriate evaluation designs. 

The evaluators trained in the evaluation centers have 
developed many models for the evaluation of 'educational programs • 
The wide variety of general purpose and specific purpose models 
that have been developed since the passage of ESEA suggests a 
framework for the cooperative use of these models, or parts of 
these models, is needed. No si|ch framework or taxonomy has been 
attempted in recent years* This paper is an attempt at the 
development of a taxoncftny of evaluation models for use in evalu- 
ating educational programs. 

Oblective ^ 

The purpose of this paper is to present a framework or ^ 
taxonomic representation of existing educational evaluation models. 
The models considered in this pape^ are those based on the 
Stufflebeam - Phi Delta Kappa National Study Comjnittee on Evaluatio 
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definition of education evaluation as the process of delineating, 
qbtaining, dnd providing useful information for judging decision 
^alternatives, 

♦lethodology 

The methodology used by the author consisted of a search of 
the literature pertaining to educational information needs and 
evaluation models axid, based'on the literature, comparisons of 
these models. Many similarities were noted in all models. It 
was also noted that several models were strong in different, areas, 
and that, depending on the objectives of the evaluation being 
•^conducted, two or more of the models could be combined to accomplish 
the evaluation more effectively than the use of a single model* 

Results of the Literature Se arch 

^ — - ■ - I ■ ■ — - r- 

In 1973, Sar' M, Steele^ identified six groups or types of 
program evaluation approaches. These were: (1) evaluation as 
input into decision-making, (2) evaluation of program parts, 
(3) evaluation— kinds of data and types of activities, (4) evalu- 
.ation processes,-^ (5) results — attainment of objectives, and 
(6) results — evaluation of outcomes and effects. Since the scope 
of this study is limited to models using the Stufflebeam - PDK 
definition, only the models identified i,n^Steele's f-'rst category- 
evaluation as input ir**:o decision making — were considered. These 
included (1) the Context-Input-Process-Product (CIPP) model 
developed by Stufflebeam; (2) the Differential Evaluation Model 
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developed by Tripodi, Fellin, and Epstein; (3) the Descrepancy 
Evaluation Model (DEM) developed by Provus; (4) Developmental 
Evaluation Models such as the IPI Formative Evaluation Model 
developed by Lindvall and Cox and the NewStart Evaluation System 
developed by Lamrock, Smith, and Warren; (5) the Priority Decisions 
Model developed by Boyle; (6) Materials Evaluation Models such as 
the Trade-Off and Comparative Cost Approach developed by Glass and 

♦ ;. ' . ' 

the Weighted Criteria Approach developed by Crane and Abt; and 

5 

(7) the Participant Reaction Approach developed by Steele, 

In addition to the list provided by Steele, the researcher 
has identified several mpdels each of whose purpose is the provision 
of information for the decision-making process. These models 
are (1) a Cost Effectiveness Model developed by Alkin,^ (2) a 
> Decision Oriented Classification Schema developed by Alkln and 
Wooley,^ (3) a Comprehensive Management Model for Title III 

8 

• Projects developed by the Fairfax County (Virginia) Public Schools, 

9 

(A) an Ontological Evaluation Model developed by Peper, (5) a 

Systems Approach Model developed by Yost and Monnin,^^ (6) the Apex 

Model developed by Morgan^"''"'' (7) Cost Utility Models developed by 

Costa and Tanner, "''^ (8) the Syiiergistic Model developed by Hunter 
13 

and Schooley, (9) the New Eclectic Model developed by Alkin 
14 

and Kosecoff , and (10) A Taxonomy of Admiiiistrative Information 
Nefeds developed by Ott (The Ott Model). 

The preceding list is not intended to be complete. It is 
intended to be representative of the models that are based on the 
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Stuf flebeam - Phi Delta Kappa National Study Committee on Evalua- 
tion's definition of Educational Evaluation as the Process of 
delineating, obtaining, and providing useful information for 
judging decision alternatives* In short, those models that were 
developed for input into the decision process. 

For the purpose of constructing a Taxonomy of Evaluation Model 
each of the models identified as a general purpose model was 
evaluated according to criteria adapted from Kirby's un- 

published Ph.D. dissertation at the University of Illinois, 
"An Approach to Decision Making." These criteria are: 

1. The model should assist the evaluator in anticipating 
all information needed for the decision process. 

2. The model should be internally logical and complete. 

3. The model should be of sufficient clarity so as to allow 
implementation by a trained evaluator without external 
interpretation. 

4. The model should relate elements in a way in which they 
have not previously been related. 

5. The model should be heuristic. 

6. The model should be capable of being extended by 
empirical study. 

7. The model should be efficient. 

The models identified as general purpose models were: the 
CIPP Model; the Differential Evaluation Model; the Discrepancy 
Evaluation Model; the Decision-Oriented Classification Model; the 
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Comprehensive Management Model; A systems Approach; and Apex - 

Model; the Synergistic Evaluation Model; and the Ott Model. 

The forerunner of all of the ^modern educational evaluation 

and information needs models is the CIPP Model developed by < 

Dr. Daniel Stuff lebeam of the Evaluatio^enter of The Ohio 

State University arid later refined by thfe Phi Delta Kappa Study 

Committee. This model was constructed to help meet the evaluation 

requirements of Title I and Title III of the Elementary and 

Secondary Education Act of 1965 and to pro.vide a means by which 

* 

information about a- particular program, or group cf program^, 
could be supplied for the decision-making process. 

The Differential Evaluation Model was developed. by Tripodi, 
Fellin, -and Epstein to provide a framework by which administrators 
of health, education, and welfare programs could obtain information 
for determining the effectiveness, efficiency, and accomplishments 
of these programs. The process, as ^developed in this^ model, was 
based on the three stages of program devet^jt^ht - program ini- 
tiation, program- contact, and program implementation - and sought 
to delineate objectives appropriate to these stages and then to 
match evaluation techniques to these objectives. ^ 

The Discrepancy Evaluation Model was developed by Provus to 
provide evaluative information for program improvement, maintenance 
or termination by defining program standards, determining whether 
discrepancies exist between these standards and actual performance, 
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and using the discrepancy information to change performaAce or^. 
standards. Provus divides tjre evaluation techniques applied to 
ongoing programs into design, 'installation, process, product, stages and 
an optional comparison, stage. Discrepancies are determined by 
examinationr- of three content categories (input, process, and'Oiitput) 
at each of the stages and by comparison of the program performance 
information obtained with previously defined standards at each 
stage. The nature and severity of the discrepancies found would 
lead the decision makers to, decide to (1) terminate the program; 

(2) make adjustmeitts in the previously designated standards; or 

(3) change the^ program performance. ♦ 

The Decision Oriented Classification Model was developed by 

t 

Alkin and Wooley to assist evaluators in (1) determining decision 
areas of .concern; (2) selecting appropriate information; and (3) 
collecting and analyzing information for use in the decision 
process. AUtin and Wooley identified five areas of evaluation that 
yield useful information for the^ decision process. These areas 
are systems assessment^ program planning, program implementation," 
program improvement, and program certification. 

The Comprehensive Management Model was developed by the 
University of Kentucky's Research Foundation for the Center for 
Effecting Change of the Fairfax County (Virginia) Public Schools. 
The model was developed to assist in the planning and evaluation 
of the Second National Study of Projects to Advance Creativity in 
Education (PACE). Like the CIPP Model, this model was constructed 
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to apply to every phase of a project ^from* Inception to termination. 

Speclfi::ally, the model itself "is designed to -^ssi^st in 'all phases 

o = ' 

of a pACE project. *As such, it could be useful for "(a) those 
'who write proposals, (b) those who operate projects, and (c) those* 

who evaluate projects in terms of general improvement and in 

16 ' * 

terms of continuation."*^ % 

Tjie Systems 'Approach was developed by Yost and Monnin^to 

provide a Specific model for the evaluation of programs funded under 

Title III ^f the Elementary and Secondary Education Act^ of 1965. 

Drawing heavily on the CIPP model ,^ the Systems Approach w€S 
constructed to include delineation, collection, interpretation, 
and distribution activit;^ps a^ each of three stages (baseline, 

, process, and product) appropriate to the general evaluation of 
Title III projects. , ' ' . . 

The Apex Evaluation Model was developed to evaluate the North 
Carolina Exemplary Program, funded under Part D of the Vocational 
Education Amendment of 1968 (P*L. 90-57^ Section 141). The.model, 
presented by Morgan, was based on: (1) the intent of legislation; ^ 
(2) the program ol?jectives (desired outcomes); (3) the process 
objectives (desired processes); (4) the observed processes (op- 
erationar procedures and resources); and (5) the output of the 
program along with the static interrelat?ionships between these 
components * 

The Synergistic Evaluation Model was developed by Hunter and 
Schooley to satisfy the need .for united efforts between curriculum 
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developers ani researcHers. The use of this model attempted to^, 
provide interaction between these experts in conducfting meaningful 
research for input into administrative decision making. To . ' 
accomplish this ^ask. Hitter and,&chooley conceptualized an edu- 
cational system as consisting of four -domains:, policy; prograra 
development; instruction; and feedback,^ and^ the corns t it uetits and ^ 
clients of the system 4s falling into six exhaustive but not*." ^ 
necessa'fily mutually exclusive groups: students-, instr^tidqaL staff ; 
administrative staff; e<|ucatiefnal specialists; family; aod ^ 
comrnynity/ Input from' all of these 'groups, according^ to the 
authors, must occur Within each of the four domains of the evalua- ^ 
tion model. ^ - - . - v ^ \ > ^ 

- s 

The Ott ^odel, A Takojlomy-- of Administrative Information 
Needs: An4id to^Edticational Planning: and 'Evaluation, is an 
outgrowth- Of Ott's earlier work "A Decision Process and Clas^lfica- 
tion System for Use in Planning Educatiot^ Chancer"' In this' 
earlier paper, Ott conducted" an empirical study o£ projects in% 
Columbus, Ohio, that were funded under Title I of the Eletnentary- 
and Secondary Education Act of 1965, In this study^^ the decision , 
process was defined as beginning with a recognition of 'the.need 
or potential need for an alternative and as endfng with the 
implementation of fhe chosen course of action. ' - ^ 

To .develop. his Classification System of Decision .Situations , 
^tt observed the decision process used by the Title I projects and 
those areas of thja projects -in which decisions were most frequently 
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made. These frequent decision areas were determined, to be 
decisions relative to Target, General Policy » Objectives, Program, 
Resources, Schedules, and Program Policy^ Ott refined the Schema 
*of the decisi9n process and combined this refinement with the areas 
in which decisions must be made most frequently to develop the 
taxonomy for administrative information needs. 

The results of,*he author's previous work in^icajted that the ^ 

Ott model, by its construction to be a logical and viable starting 

/ . < 17 ' 

.point for the development of a taxonomy of evaluation models* 

■-^herefore,*the author compared each of the general purpose models 
considered abo^ with the Ott Mpdel on the previously identified 

^evaluati"V{e ^criteria, 

' Table 1 suramarizec the results of the comparisons between the 
Ott Model and each of the general purpose models f-ound ^^the 
literature. The table .indicates those criteria on which the Ott . 

, Model was superior (+) , inferior (-) , and about equal (=) [ to 
the model under consideration. ' . * ^ 
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Tl^ remainder of the models and their identified specific^ 
purposes were: The IPI Formative Evaluation Model aad the NewStart 
Evciluation Sy^'tem f'>r the evaluation of instructional and indi- 
vidualized .ulum packages; the Priority Decisions Model for 
determining program priorities; the Trade-Off and Comparative 
Cost Approach and the Weighted Criteria Approach for the evalua- 
tion of educational materials; the Participant Reaction Approach 
as evaluative input into program modification; the Cost Effective- 
ness Model and the Cost Utility Models for evaluation of programs 
in terms of program costs; the Ontological Evaluation Model for 
evaluation of the organization in which a program functions; and 
the New Eclectic Model for framing the decision context of a 

a. 

program evaluation. 

The IPI Model uses formative evaluation, individual pupil 
monitoring techniques, and summative evaluation to assess the goals 
that the program should achieve, the plans for achievement of 
these goals, the amount of agreement between the actual program 
and the intended program, and the achievement of the intended 

goals, Fonua^tJLve evaluation is also- used to insure that each of 

the steps in the program or package is accomplished with care 
and quality. Weaknesses, if discovered at any step,' provide 
information for modification of preceding steps. 

Lamrock, Smith, and Warren use formative and suinraati^ye 
evaluation activities, in the NewStart Model, to get information 
" "that will help users in installation of developed and tested 



course packages. This information was based on the topic areas 
where decisions are made' (policy level; types of intervention, 
dissemination, and organizational structure; methods and systems 
of evaluation and measurement; the theory and models of training 
practices^ student progress; ability, style, personality and train 
ing of tutors; and curriculum content), the type"* of databases 
used in the decision, the source of the data for the "tfecision, and 
the timeliness of the decision. 

The Priority Decision Model was developed by Boyle to help 
decision makers determine program priori-^ties These techniques 
ar^ especially useful when administrators are faced with too much 
content to taaCh, too many problems to solve, or too many clients 
to reach and serve effectively. 

The Trade-Off and Comparative Cost Approach is based on 
practical and prudential considerations for examining materials 
for possible adoption. The actual areas presented for considerati 
by the model are: (1) product description (to provide a detailed 
description of the product under consideration); (2) goals evalua- 

t^on--(4:^k©termin&. the _^Ojals^ PJ---J^e._prodhuct and their appro- 

priateness); (3) clarification of point of entry of the evaluator 
(to determine the reversible and irreversible decisions to be 
served); (4) trade-offs (to determine the kind^ of trade-offs, 
the alternative methods, and the possible effects 'of the trade- 
offs); (5) comparative, cost analysis (to exaaine the product 
cost and compare the cost with the cost of alternative products); 
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(6) intrinsic evaluation (to assess the technical quality, content, 
and uniqueness o^'f the medium Involved, and to survey the avail- 
ability of resources needed for use of the product); (7) outcome 
evaluation (to assess the learning rate Bnd methods used, the 
knowledge acquisition and rationale used, and the retention of 
knowledge); (8) summative judgments and recommendations (to judge 
the quality and effectiveness of the product and to make recoramen- 
dations for potential consumers, current developers or sponsors); 
(9) circumstances modifying the summative judgments (to examine 
the scope of the value claims and to test conditions that could 

cause modification of the recommendations); and (10) evaluate 

• •> 
the evaluator (to explore possible motives, biases, and considera- 

tions that may have influenced the evaluator)^ 

The Weighted Criteria Approach was founded on cost effec- 
tiveness techniques and then applied to curriculum materials to 
analyze components, quality, and cost of the product .or material 
under consideration. The comppnents of the model are weighted 
by relative importance* The components of the model are: (1) 
coverage (in terms of scope and^quality) ; (2) appropriateness; 
(3) motivational effectiveness (for both the student and the 
teacher); and (4) cost (in terms of dollar amounts and time). 
The Weighted Criteria Approach can be used to assess relative 
merit of two or more alternative materials. 

The Participant Reaction Approach was developed' by Steele 
specifically to evaluate adult education and extension courses. 

16 



Steele felt that these types of programs, being more scattered 
than centrally located, would be more easily evaluated by seeking 
the reaction of the clients served by the programs. To accomplish 
this, Steele employs a vaiciety of techniques such as steering 
committees, analyses of attendance and reenrollment records, and 
end-of-session reaction forms • The information gathered in this 
manner is intend^ for use in making decisions about program 
modifications. 

The cost Effectiveness Model vas developed by Alkin based 
on the techniques of cost-benefit analysis. Aft^ir reviewing the 
shortcomings of cost-benefit analysis, Alkin sought to present 
eval :ation techniques capable of consideration of all relevant 
elements of an educational system (at the school district level) 
to: (1) compare educational^ outcomes of different units; 
(2) assess the impact of alternative levels of financial input; 
and (3) select alternative approaches to the achievement of 
specified educational outcomes. 

Costa undertook the , study and presentation of Cost Utility 
Models to provide viable forms of cost effectiveness - cost 
benefit analysis evaluation techniques for use by existing school 
staff. The information resulting from the application of th^se 
evaluation models was intended to provide input into the decision- 
making process in the school system under consideration. Costa 
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presents two models (Tanner's !'odel developed by Tanner and the 
Milwaukee Model developed by Costa and Giroux) that (1) can be 
used by administrators and their staffs to choose between alterna- 
tive programs; (2) use relatively easy computational formulas; 
(3) lack confusing "jargon"; and (4) are able to provide valuable 
rational decision-making information. 4 

The OntologicaJL Evaluatit>n Model was developed by Peper to 
fill what he considered to be a void in the then existing 
evaluation models - a model that would focus on the organization 
as a whole and not on a single program withia the organization. 

In constructing this model, Peper conceptualized two areas of 
organizational de^^elopment: organizational characteristics and 
phases of organizational development. Organizational character- 
istics are morphology or structure, systems, kinesthesia or thrust, 
and kinship linkages. The phases of organizational development 
are preemergent, emergent, familial dependent, familial independent 
and independent. 

The New Eclectic Model whs developed by Alkin and Kosecoff 
to provide a means of putting together the "old but stili good 
^Idea^^itT" eclectic or" unique ways. Thtrs-irhe nodet-seeks-^nt^e-the - 
best evaluation models and ideas selected from existing sources to 
structure the evaluation process. The actual model requires four 
steps or procedures: (1) framing the decision context; (2) 
building the evaluation system; (3) determining techniques and 
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methodologies; and (4) preparing and reporting evaluation informa- 
tion. Believing that the last three areas presented above are 
those which lend themselves easily to the "old .but still useful 
ideas," Alkin and Kosecoff stress only the step for framing the 
decision context. 

Results 

Before presenting the framework for the cooperative use the' 
general and specific purpose models discussed previously let us 
examine the Ott Model more closely. 

To develop his Classification System of Decision Situations, 
Ott observed the decision process used by the Title I projects 
and those areas of the projects in which decisions were most 
frequently made. These frequent decision areas were de^termined 
to be decisions relative to Target, General Policy, Objectives, 
Program, Resources, Schedules, and Program Policy. Ott refined 
the Schema of the decision process and combined this refinement 
with the areas in which decisions must be made most frequently to 
develop the taxonomy of administrative information needs. 

The Ott Model is presented in Fi gure 1. The areas in vhi^^ 

decisions must be made most frequently (Target, General Policy, 
Objectives, Program, Resources, Schedules, and Program Policy), 
* comprise the first dimension of a two-dimensional framework in 
the Ott Model. The second dimension consists of the stages of the 
decision process for which the information is needed. These stages 
arc: (1) Recognition of the Present of Potential Problem, 
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(2) Definition of the Problem, (3) Establishment of a Probable 
Cause of the Problem, (4) Establishment of Criteria for Judging 
Alternative Innovations, (5) Exploration of Alternatives, (6) 
Design of Potential Innovations, (7) Trial, (8) Decision Point, and 
(9) Implementation. 

The major program evaluation emphasis of the Ott Model is 
found in the first stage of the decision process — the Recognition 
of the present or Potential Problems. It is this stage which 
fully examines the areas of the program in which decisions are most 
frequently made. The key question and information needed for each 
area of concern within this stage are categorized below. 

Target: What are the characteristics and needs of the 
target populatioi|? Are these needs taken into consideration? Is 
the appropriate student included in the program? Are there incon- 
sistencies between the needs of the target population, the student 
and community needs, and the project capabilities, priorities'; and 

" responsibilities? 

General Policy: What are the general guidelines for the 
project? What general policies of the parent system affect the 

__j:arget_pQpulation? Are_-thex£.^y_.JLiicoiis,la.t_encies_bet^^ 

guidelines for^the project and the general policies of the parent 
system? 

Objectives: What are the project or program objectives? 
Do the objectives reflect the needs of the identified target popu- 
lation and the capabilities and priorities of the project? 



Program: What are the characteristics of the program 
involved? there inconsistencies between the program and the 

desired solutions or outcomes? Is the program actually accom- 
plishing what it intended? Is the program efficient? 

'Resources: Wliat resources are needed? What resources are 
being used? Is the project making the best use of resources at 
its disposal? Id there an efficiently functioning support system 
for resource supply and maintenance? 

Schedules: Can the project accomplish its intended objectives 
in the time specified? Are there internal and/or external in- 
consistencies that will affect the accomplishment' of the project.- 
objectives according to the stated schedule? 

Program Policy: \^at are the project guidelines for action 
within the proj^ect? Are there inconsistencies between the project 
action and the needs of the target population? 

Any discrepancies and inconsistencies in data that are found 
during the process of answering these questions will provide an 
indication of a present or potential problem. When this occurs, 
the process continues through the following stages. 

The Definition of the Problem is the stage at which the in- 
consistencies' are described. Information needed here includes the 
details of the present and potential effects of the problem with 
supporting evidence such as past experiences with similar problems, 
similar experiences of others, and speculation of those closely 
involved in the problem area. 



' Sta^e III ; The Establishment £f Probable Cause of the Problem 
Ijicludes a description of possible causes of the problem. In- 
formation needed includes supporting evidence such as pasf^ex- 
pexience or expe9fehce^ of others, simultaneous ^vents, theoreti- 
cal relationships, and speculation of those affected. 

Stake IV . The Establishment of Criteria for Judging Alterna- 
tive Innovations includes specification of the effects wanted and 
unwanted by those responsible for the project and deciding upon 
effects that should be sought or avoided. Information is needed to 
establish the relative importance of possible effects according to 
those affected, those responsible for the project, and theory or 
experts. . Rationale for this information should also be included. 

Stage V . The Exploration of Alternatives Stage indicates how 
others have dealt with similar problems. Information needed at 
tthis stage includes the: suggestions of those concerned or involved 
with the program, and alternatives suggested by experts or the 
literature. • "^-^ 

Stage VI . The Design of the Potential Innovations , draws 
upon the e:?p:eri^nce of others, and examination of the outcomes^ of 
similar innovations. The informa^tion needed includes the predic- 
"tTdTr"Ot^expet^a-^ancarrdjut.oi ^ d escription of available 

resources, (time, human, and material), and description of relevant 
portions of the sys.tem in which an innovation must be made. The 
rationale for the above information should also be included. 

Stage VII . The Trial Stage includes a description of the pro- 
posed innovation and alternate ways of testing' the proposed in- 
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novaPtibn Including the requiraments of, and "pay-off" for the » 

alternative modes of testing the proposed innovation. Information 

needed for this stage includes a description of the system in which 

the test is to be made, a description of the trial, the incon-* 

sistencies between the trial and the system in which it is to be ^ 

installed, the problems of *the implementation and trial processes 

and their probable causes and effects, and alternative corrective 

measures and their probable effects*. 

Stage VIII * The Decision Point Stag^e compares the ijerformance 

of the potential innovation on important criterion variables, both 

anticipated and unanticipated. Information needed includes 

^ • ' . . ^ - • 

comparisons of potential innovations with other courses* of action, 

and the problems of design; implementation, and functioning 

revealed in the trial. ' \ 

Stage IX . The Implejnentation Stage includes a description of 
the innovation, the system or subsystem in which it is to be In- ' 
stalled, and any inconsistencies between the innovation 'and the 
system. Information needed here includes alternative strategies 
for resolving the inconsistencies and the effects of thcsfe al- 
ternatives, requirements and pay-offs for each strategy, problems 
of J.mplementation and their probable causes, and outcomes of the 
implementation and corrective measures. 

It should be noted at this point t}\at the evaluation pro-, 
cess and the information gathtiring process are continuous pro- 
cesses. As such; information gathered at aay subsBquerft stage 
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can bring to light problems that were not uncovered before. In 

fact /new information can swerve to, relnfc^rce or change a previously 
^< 

made de^cision. ^ . ^ ^ 

Now that the Ott Model* has been presented, how can we use 

thQ previously presented, genaral purpose and specific purpose ^ 

models'^ with*' in the Ott Model for the effective and efficient eval- 

uation of educational programs,? 

^Thia^author believes that before any program evaluation is 



conducted the purpose of the evaluation* and the decision makers 
should be identified. This is not represeijfed eAplicitly in the 
ott Model. /The ^ut.hor suggests that the Alkin and Kose^off 
Modef, "A New Eclectic Model for Redirection of Evaluation 
Efforts"^ be used to frame the decision context. This entails the 
conside^lfcion of the decision maker (s) and/or decision audiences, 
the explicit decisic*fi-making .pur|io&es , the implicit decision-^maklng 
purposes, the developmental stage of the progjram to be fevaluted, 
and the socio-political setting of tl^ evaluation to be cojiducted. 

Once the evaluator determines the decision context, he is ♦ 
ready to begin the problem recognition stage ^-^of the decisrion process. 
This stage is the part of the Ott Model. that is most similar to 
the already presented general purpose models and forras^ the nucleus 
of the evaluation of afi educational program. From" Figure 1, it is 
^obvious that this is also the" most detailed part of the ittiodel. '^^^ ^ 
areas of qoncern to decision makers, suggested information, and 
general purpose and specific models that could be used to 
provide information -needed for decision makers is listed bel9w. 
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The reader will note that use of the Ott Model requires 
background information about all areas of concern to decision 
makers. If this is not available or 'complete, the evaluator can 
use needs assessment techniques presented in the needs assessment ' 
section of the proposal development phase of the Comprehensive 
Management Model, or the systems assessment phase of the Decision 
Orient?jd Classification System. 

- '^o collect information for the recognition- of problems dLn the 
area of the target population, the evaluator could use the techniques 
of the Discrepancy Evaluation Model, the Differential Evaluation 
Model, or the Priority Decisions Model. Information for the recog- 
nition of problems in the area of general policies could be 
gathered by a'se of thi Participant Reaction Approach, the Priority 
Decision Model, and ^the Discrepancr Evaluation Model. In the area 
of objectives, the evaluator could use the Participant Reaction ' 
Approach or the techniques of the Discrepant/ Evaluation Model, 
.inconsistencies between the actual outcomes and the desired 
outcomes .would be determined bythe "traditional" statistical 
analysis techniques. In the area of -resources, information could 
be provided by the evaluator 's use of the Participant Reaction 
Approach and the Descrepancy Evaluation Model. Infomation about • 
schndule conflicts could be provided thrpugh the use of the " 
Participant Reaction Approach. Finally, the Priority Decision 
Model and the Participant Reaction Approach r-r jld be vised to pro- 
vide information c^out the program policy and the established 
patterns of action within the program. 
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When the evaluator, through analysis of the data collected 
in the .first stage, recognizes the existence of a present or 
potential problem, he initiates the implementation of the sub- 
sequent stages of the decision process. • The exact techniques and 
methM used to gather information at each of 'these stages depends 
on the nature of each of the problems identified. Some of the 
general and specific purpose models that could be used are pre- 
sented below. 

At the problem definition stage, the description of the 
problem itself t,hould be a result of the analysis of the information 

i 

gathered in the previous stage. Details of actual and/or po- 
tential effects of this problem^and supporting evidence, could 
be gathered by use of a literature search of past experiences of 
other persons with similar programs. 

In the third stage. Establishment of Probable Cause of 
Problem, the description of possible causes of the problem (with 
supporting evidence) could be determined by statistical analysis 
Participant Re actiorrl^pp roaches, and literature techniques.. 

For stage 4, Establistiment of Criteria for Judging Alter- 
native Innovations, the, necessary information for determining the 
effects wanted or unv^anted, and theii relative importance, can 
be supplied by using the procedures and techniques designed to 
evaluate the needs and alternative ideas in the Priority Decisions 
Model* 

In the Design of the Pot^ntidl Innovation ^age (Stage 5) the 
information needed by decision makers includes how others have 

': ^ ■ " ' 
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dealt with similar problems, suggestion of participants, and 
alternatives suggested by experts or professionals. This informa- . 
tion can be provided through the Participant Reaction Approach and 
through a search of the relevant literature. 

In the Design of the Potential Innovation, (stage 6) the 
evaluator needs to supply information to the decision makers about 
experiences of others with similar innovations, potential 
outcomes, descriptions of available -resources, and descriptions of , 
the relevant portion of the system In which the innovation Is to 
be made. This information can be provided by usin^ the Develop- 
mental Models (such as IPI and NewStart techniques), Material 
Evaluation techniques (such as the Trade-off and Comparative Cost 
Approach or the Weighted Criteria Approach), and the Cost- 
Effectiveness and Cost Utility Models. 

' At^the Trial Stage (Stage 7) the innovation or alternative 
innovations are te^sted for their suitability for eliminating the . 
problem. The Trade-off and Comparative Cost Approach would pro- 
vide information about alternative ways of testing the innovation. 
The Ontological Model could provide a description of /the system 
in which the test is, to be made and the inconsistencies between 
the proposed trial and the system in which it is toCbe installed* 
The IPI and NewStart Models could be used to reve il problems of 
implementation, and to identify corrective measures to be considered. 

At the Decision Point (Stage 8) the results of the Trial 
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Stage are considered. Information is needed here on the perfor- 
mance of the innovation on important criterion variables, on 
comparisons of the potential innovations with other courses of 
action, and on problems of design, implementation and function 
of the innovation that were revealed in the trial. This information 
can be provided by analysis bf the inforination gathered at the 
trial stage. 

In the Implementation Stage (stage 9) information relative to 
the proposed innovation, thp sub-system^in which it is to be 
installed, .actual or potential inconsistencies bet\teen the in-- 
novation and the subsystem, alternative strategies for resolving 
these inconsistencies, potential implementation problems, and 
possible outcome of the implementation Js needed. This informa- 
tion can be provided by the Developmental and Materials Evaluation 
Models, the Ontological Evaluation Model, the Discrepancy Evaluation 
Model, and the techniques outlined in the Differential Evaluation 
Model. 

In summary, it is ,the author's belief that the use of the Ott 
Model "A Taxonomy of Administrative Inforination Needs: An Aid to 
Educational Planning and Evaluation" as an overall guiding 
framework for tjtie cooperative use of the many general purpose and 
special purpose evaluation models found in the literature will 
provide the necessary evaluative information needed by decision- 
makers. The use of this framework will provide more effective and 
efficient program evaluation in public education. 
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FOOTNOTES 

^Daniel L. Stuff lebeam, "Evaluation as Enlightenment for 
Decision Making," (Columbus, Ohio: The Evaluation Center, The 
Ohio State University, 1968), p. 5, (Mimeographed). 

2lbid. p.- 8. 

3lbid., P. 16. 

^Sara M. Steele, "Contemporary Approaches to Program 
'Evaluation and Their Implications for Evaluating Programs for 
Disadvantages Adults" (Syracuse: ERIC Clearinghouse on Adult' 
Education, 1973) (Microfiche ERIC ED 075 715). 

Hhid., p. 51. 

Marvin C. Alkin, "Evaluating the Cost Effectiveness of 
Instructional Programs," Proceedings of the Symposium on Problems 
.in the Evaluation of Instruction , (Los Angeles: University of 
^California) (Microfiche ERIC ED 031 8l'8).. 

^Marvin C. Alkin and Dale C. Wooley, "A Model for Edu- 
cational Evaluation" (paper presented to the PLEDGE Conference, 
*San Dimas, California, October 8-11, 1969) (Microfiche ERIC ED 
036 898). ' • > 

^F^irfax County (Virginia) Public Schools, Center for 
Effecting Educational Change, "A Comprehensive Model for^ Managing 
an ESEA Title III Project from Conception to Culmination" 
(Micro-fiche ERIC ED 039 278). 

^John B. Peper, "An Ontological Model of Evaluation: A 
nynamic Model for Aiding Organizational Development" (paper pre- - 
sented at the AERA Annual Meeting, New Orleans, February 26 - 
>Iarch 1, 1973) (Microfiche ERIC ED 078 039). 

lOMarlen Yost and Frank J. Monnin, "A Systems Approach to 
the Development of an Evaluation System of ESEA Title III 
ProjectsV (unpublished research paper) (Microfiv.he ERIC ED 047 356) . 

^^Robert L. Morgan, "An Approach to Evaluation: A Model 
f6r Evaluation of the North Carolina Exemplary Program" (Raleigh: 
North Earolina State University, 1970) (Microfiche ERIC ED 042 9J.0). 

l^crist H. Costa, "Cost Utility: An Aid to Decision Making" 
(paper presented at the AERA Annual Meeting, New Orleans, February 25 
March 1, 1973) (Microfiche ERIC ED 074 623). 
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Michael G. Hunter and Daniel Schooley, "The Synergistic 
Evaluation IJodel*' (paper presented at the AERA Annual Meeting, 
Nev7 Orleans, February 25 - March 1, 1973) (Mimeographed). 

■'•^arvin C, Alkin and Jacqueline B. Kosecoff , "A New 
Eclectic Model for the Redirection of Evaluation Efforts" (paper 
pres~ente4 at the AERA Annual Meeting, New Orleans, February 25 - 
March 1, 1973) (Mimeographed). 

15 

Jack M. Ott, Sheila Fletcher and Donald Turner, "A 
Taxonomy of Administrative Information Needs: An Aid to Educational 
Planning and Evaluation," Educational Technology » May, 1973. 

■^^Fairfax County (Virginia) Public Schools, Op cit., p. 5. 
1 7 

7* Wayne E. Carter, "An Evaluation of an Evaluation Model," 
(paper presented at thfe AERA Annaul Meeting, Chicago, April 16, 197A) 
and Wayne E. Carter, "A Taxonomy of Administrative Information 
Needs: An Evaluation of the Model," Unpublished Ph»D. dissertation, 
College of Education, University of South Carolina^ 1974. 
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